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Proposed Methodology for Defining Learning Gains 


At a Glance 
This report examines one component of the current school accountability system - 
learning gains. We find that learning gain percentages are affected by a variety of 
factors including starting achievement level, socioeconomic status, and English 
language learner status when the current definition of learning gains is used. Hence, 
we propose a new methodology for defining learning gains in terms of scale score 


increase targets. Under this new definition, students who are behind their peers in 
academic achievement will be required to demonstrate higher scale increases to meet 
the target for learning gains. We find that the proposed definition of learning gains 
reduces the variability in percentages of students making such gains depending on 
their initial achievement results. In addition, it reduces the gap between the learning 
gain percentages for certain accountability subgroups. 


In January 2020, Governor DeSantis announced the creation of new State academic standards, which will 
be called the BEST (Benchmarks for Excellent Student Thinking) Standards. The published timeline 
anticipates the implementation of the BEST English Language Arts (ELA) standards in 2021-2022, and 
those in mathematics — in 2022-2023. The implementation of the new standards will require the creation of 
new assessments aligned to these standards. The implementation timeline also states that the school year 
2021-2022 will be the last one in which the Grade 9 ELA assessment and Geometry EOC assessment are 
administered. The elimination of these assessments will necessitate changes to the State’s school 
accountability system. 


The current school accountability system has many components, one of which is the percentage of students 
making learning gains. In this Research Brief, we propose a new method for defining what constitutes a 
learning gain. 


The Current Definition of Learning Gains and its Implications 

The results of the Florida State Standards (FSA) testing in ELA and mathematics are reported in terms of 
scale scores and Achievement Levels ranging from | to 5. A student scoring within Achievement Levels 3- 
5 is considered achieving “at grade level”. The Achievement Levels are defined in terms of specific ranges 
of scale scores. For example, in Grade 3, the interval of scale scores ranging from 240 to 284 in both ELA 


and mathematics define Achievement Level 1. Because the FSA scale is constructed to be vertical', the 
endpoints of the range defining a particular Achievement Level increase from one grade level to the next. 
For instance, the interval ranging from 251 to 296 define the Achievement Level 1 for Grade 4 ELA results, 
while the interval from 251 to 298 define that for mathematics. 


To define learning gains, the scale score ranges defining Achievement Level | in each grade and subject 
are subdivided into the three smaller sublevels of approximately equal width (Level 1 Low, Middle, and 
High), while the ranges for Achievement Level 2 are divided into two sublevels (Level 2 Low and High) 
as shown in Table | below for ELA scores. 


Table 1 
FSA ELA Scale Score Ranges for Learning Gains 


Grade__|LiLow_|L1Middle|L1High |L2Low [12 High | Level 3 | Level 4 | Level 5 


According to the current definition of learning gains for a student who has assessment results in the same 
subject in two consecutive school years, the student can demonstrate a learning gain in several ways: 


e Advance by at least one Achievement Level, 

e For students who score within Achievement Level 1 in both years or Level 2 in both years, 
advance at least one sublevel, 

e For students who score within Achievement Level 3 in both years or Level 4 in both years, 
increase a scale score by at least one point, or 

e Remain within Achievement Level 5 in both years. 


Notice that a student can demonstrate a learning gain when advancing by as little as 4 scale score points 
(from 272 in Grade 5 to 276 in Grade 6) or even when losing as much as 33 scale score points (from 403 
in Grade 8 to 370 in Grade 9). Figure 1 shows the percentages of students across grades 3-9 making learning 
gains in ELA? from 2018 to 2019 across all Grade 3-9 by students’ initial Achievement Level or Sublevel. 


‘Tn an ideal vertical scale, equal increases in scale scores signify equal amounts of learning across time. 
> In the rest of the document, we limit examples to ELA results of students who advanced to a next grade level from 
2017-2018 to 2018-2019 school year. 
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Figure 1. Percentages of Students Making Learning Gains from 2018 to 2019 


Clearly, the percentage of students making learning gains had a large variability ranging from 43% of 
students who were in Achievement Sublevel 1 High to 69% of students who were in the Achievement Level 
3 in 2018. 


To illustrate the idiosyncratic consequences of the learning gains definition further, we used the data from 
students who went from Grade 3 in 2018 to Grade 4 in 2019, as an example. We split the range of scale 
scores in ELA into 20 subintervals so that each interval corresponded to roughly 5% of students who scored 
within this interval on the 2018 FSA ELA. Each interval contained about 1,000 students. We then 
determined the percentages of students making learning gains for each of these 20 subintervals; the results 
are shown in Figure 2. 
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Figure 2. Percentages of Grade 3 Students Making Learning Gains from 2018 to 2019 by 2018 Scale Score 
Interval 


The variability of the learning gains percentage shown in Figure 2 is rather large ranging from 42% to 82% 
demonstrating the differential impact of the definition of learning gains depending on the initial scale score. 
The percentages of students making learning gains were much lower for students who scored within 
Achievement Levels 1-2 on the 2018 FSA ELA assessment (shown to the left of the vertical line in Figure 
2) than for those who scored within Achievement Levels 3-5. 


Because following the current definition of making a learning gain it is easier to demonstrate learning gains 
for students who score proficient (within Achievement Levels 3-5), there is a differential impact on student 
subgroups in terms of proportions of students making learning gains. We illustrate this impact by examining 
two accountability subgroups: economically disadvantaged students (defined as students eligible for the 
federal free or reduced-price lunch [FRL]) and English Language Learners (ELL), defined here as those 
who were in the ESOL program as of May 1, 2019, or exited the program within the 2-year interval from 
that date. The results are shown in Figure 3, which displays the percentages of students making learning 
gains (the left axis) and the mean gains in scale scores from Grade 3 in 2018 to Grade 4 in 2019 (right axis). 
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Figure 3. Percentages of Students Making Learning Gains and the Mean Scale Score Gains from Grade 3 
in 2018 to Grade 4 in 2019 by Student Subgroup 


Economically disadvantaged students demonstrated a smaller percentage of making learning gains 
compared with the non-economically disadvantaged counterparts despite making approximately equal 
gains in scale scores from 2018 to 2019. The comparison of the two outcome statistics shows that ELL 
students made considerably larger gains in scale score points than their non-ELL peers; yet, their percentage 
of making learning gains was smaller. 


Schools in which ELL and economically disadvantaged subgroups are represented to a large degree are 
likely to demonstrate smaller learning gains due, at least in part, to the disparate effect of the definition of 
learning gains on subgroups. Figure 4 shows a relationship between the percentage of FRL students and the 
percentage of students making learning gains from 2018 to 2019 for 100 randomly selected District schools. 
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Figure 4. Percentages of Students Making Learning Gains from 2018 to 2019 by School Percentage of FRL 
Students 


The inverse relationship between the school percentages of FRL students and learning gains can be seen 
clearly in Figure 4. When the data on all District schools are used, the value of the coefficient of 
determination R’ is approximately .30 indicating that about 30% of the differences in school percentages 
of students making learning gains can be explained by their percentages of economically disadvantaged 
students. The values of R’ by school type were .29, .63, .38, and .30 for Elementary, Middle, High, and 
Combination schools respectively. 


Toward an Alternative Definition of Learning Gains 

If the pattern of academic growth in ELA were similar for all grade levels, it would be possible to establish 
a common definition of learning gains across grades. This is not the case, as shown in Figure 5, which 
demonstrates that the mean changes in scale scores from 2018 to 2019 varied by the grade in a somewhat 
haphazard manner. 
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Figure 5. Mean Differences Between 2019 and 2018 FSA Scale Scores by Student Grade 


Thus, a definition of learning gains in terms of scale score gain would need to be established separately for 
different grades. We demonstrate a proposed methodology for establishing a definition of learning gains 
using an example of Grade 3 to Grade 4 growth in scores. We think that a reasonable definition of learning 
gains should require students who are not yet performing at grade level to demonstrate larger amounts of 
growth than students who are already performing at grade level. Students who are farther behind should 
demonstrate larger academic growth for it to be considered a “learning gain”. On the other hand, even those 
students who are performing on grade level should demonstrate some amount of growth with a possible 
exception of students whose scale scores are close to the maximum of the scale. 


To investigate the pattern of ELA academic growth for students promoted from Grade 3 in 2018 to Grade 
4 in 2019, we split the student population into deciles and calculated the mean amount of growth in scale 
score points and its standard deviation by decile (see Table 2). One can see a decreasing relationship 
between student scale score gains and the decile of their initial scale score. In addition, it shows 
heteroscedasticity (unequal measures of spread, such as standard deviation) of the gain scores depending 
on the initial score decile. 


3 This inverse relationship is due, at least in part, to the statistical/measurement phenomenon known as the 
regression toward the mean. 


Table 2 
Mean Growth in FSA ELA Scale Scores from Grade 3 in 2018 to Grade 4 in 2019 and their Standard 
Deviations by Student 2018 Decile 


Mean Std. 
Decile | N Growth Deviation 
1 | 2344 21 17.6 
2 | 2545 14 13.2 
3 | 2204 12 12.3 
4 | 2270 11 11.7 
5 | 2447 11 11.9 
6 | 2483 10 11.2 
7 | 2442 10 10.9 
8 | 2452 9 11.1 
9 | 2226 7 11.1 
10 | 2394 2 11.6 


We used quantile regression to calculate the expected amounts of scale score gain depending on the initial 
(2018) scale scores of students in Grade 3. Unlike standard least-squares linear regression, quantile 
regression can be used in the presence of heteroscedasticity of outcome scores. In addition, the quantile 
regression can be fitted not just for the mean outcome, but for various percentile values of the outcome. We 
fitted the quantile regression for percentiles ranging from 40" to 60" with 5-point increments. 


Although it is possible to use the results of the quantile regression analysis to establish the learning gain 
targets for each of the initial scale scores, we think this would be impractical. Student groupings, such as 
deciles, can be used for this purpose. This approach, too, might be impractical because the boundaries for 
deciles are likely to shift from one test administration to the next. Instead, we used the already established 
definition of Achievement Levels and Sublevels and then defined targets for increases in scale score points 
equal to the means of expected score growth from the quantile regression analysis (based on the 40" 
percentile). These targets, the resulting percentages of students making learning gains, and those based on 
the current rule for calculating learning gains are shown in Table 3 below. 


It can be seen that the largest difference between the percentages of students making learning gains based 
on the proposed targets by their initial Achievement Level is 9 percentage points. By way of comparison, 
the largest such difference based on the current definition of learning gains is 31 percentage points. 


Table 3 
Targets for Growth in FSA ELA Scale Scores from Grade 3 in 2018 to Grade 4 in 2019 by Student 2018 


Achievement Level, the Resulting and Current Percentages of Students Making Learning Gains 


[2018 Achievement Level! _N_| Target | 
05 


23807| 


Figure 6 contrasts percentages of students making learning gains from Grade 3 in 2018 to Grade 4 in 2019 
depending on their 2018 scale score under the current and proposed definitions of making a learning gain. 
It can be observed that the proposed definition leads to a more uniform pattern. The exception to that pattern 
occurs for students who scored at the upper range of Achievement Level 5 in 2018. A closer examination 
of these students’ results shows that Grade 3 students who scored 349 or above on the 2018 FSA ELA 
(approximately 400 such students, or about one-third of the highest scoring group in Figure 6) had an 
average of 6.4 scale score points loss from 2018 to 2019 testing. In that group, 78% made learning gains 
by the current definition despite “losing ground” on the scale score metric, but only 27% made it by the 


proposed definition. 
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Figure 6. Percentages of Grade 3 Students Making Learning Gains from 2018 to 2019 by 2018 Scale Score 
Interval 


Table 4 compares the effects of the proposed and current definitions of learning gains on the percentages 
of students demonstrating such gains for the ELL and economically disadvantaged accountability 
subgroups. 


Table 4 

Mean Gains in Scale Scores from Grade 3 in 2018 to Grade 4 in 2019 and Corresponding Percentages of 
Students Making Learning Gains by the Proposed and Current Definitions for Two Accountability 
Subgroups 


[Subgroup| Mean Growth | Proposed | Current| 


It can be seen that the differences in learning gain percentages between FRL and non-FRL students were 


reduced from 12 to 8 percentage points; the differences between ELL and non-ELL learning gain 
percentages were reduced from 2 to 0 percentage points. 


Discussion 

One of the current school accountability system components is the percentage of students making learning 
gains. Under the current system, students can demonstrate a learning gain when advancing by as little as 4 
scale score points or even when losing as much as 33 scale score points, while maintaining an Achievement 
Level 5. We found that the percentage of students in grades 3-9 making learning gains in 2019 had a large 
variability ranging from 43% of students who were in Achievement Sublevel | High to 69% of students 
who were in the Achievement Level 3 in 2018. 


The current definition of making a learning gain makes it is easier to demonstrate learning gains for students 
who score proficient (within Achievement Levels 3-5). Besides, it has a differential impact on student 
accountability subgroups: we found that economically disadvantaged students demonstrated a smaller 
percentage of making learning gains compared with the non-economically disadvantaged counterparts 
despite making approximately equal gains in scale scores from 2018 to 2019. In addition, we found that 
ELL students made considerably larger gains in scale score points than their non-ELL peers; yet, their 
percentage of making learning gains was smaller. Consequently, schools in which ELL and economically 
disadvantaged subgroups are represented to a large degree were likely to demonstrate smaller learning gains 
due, at least in part, to the disparate effect of the definition of learning gains on subgroups. 


In this report, we propose a definition of learning gains in terms of scale score gains. We used the ELA data 
for students who advanced from Grade 3 in 2018 to Grade 4 in 2019 to demonstrate how the targets for 
increases in scale score points can be defined. By using this methodology, we found percentages of students 
making learning gains were more stable across their initial achievement results and more similar for student 
accountability subgroups compared with the current definition or learning gains. 


Appendix A 


On March 13, 2020, Miami Dade County Public Schools were closed to stop the spread of the COVID-19 
(Coronavirus) pandemic. Subsequently, on March 17, 2020, Florida Gov. Ron DeSantis announced that the 
state of Florida was canceling all statewide standardized student assessment for Florida's public K-12 
schools for the rest of the 2019-20 school year. The 2019-2020 and school grades will not be issued. 


Given that learning gains are a major component in the current school accountability system, the Florida 
Department of Education will have to address the challenge of determining learning gains in 2021. We 
think that the methodology described in this Research Brief can be used to establish learning gain targets 
for two academic years, with a gap year in between, to address this challenge. 


We again used quantile regression to calculate the expected amounts of scale score gain in ELA depending 
on the initial scale scores of those students in Grade 3 through 8 in 2017 who advance by two grade levels 
by 2018-2019. We fitted the quantile regression for percentiles ranging from 40" to 60" with 5-point 
increments. We used the already established definition of Achievement Levels and Sublevels and then 
defined targets for increases in scale score points equal to the means of expected score growth from the 
quantile regression analysis (based on the 45" percentile for grades 5-9 and the 40" percentile for grade 
10). These targets and the resulting percentages of students making learning gains by grade are shown in 
Table 5 below. 


Table 5. 
Targets for Growth in FSA ELA Scale Scores from Grades 3-8 in 2017 to Grades 5-10 in 2019 by Students 
2017 Achievement Level, the Resulting Percentages of Students Making Learning Gains 


2019 Grade Level 


Grade 5 Grade 6 Grade 7 
2017 Achievement Level N Target | Percentage N Target | Percentage N Target | Percentage 
Level 1 Low 406 27 63% 813 24 57% 952 26 61% 
Level 1 Middle 904 25 63% 1047 22 52% 1263 22 57% 
Level 1 High 1904 23 57% 2638 19 57% 2828 18 54% 
Level 2 Low 2547 21 52% 2216 18 53% 2688 15 55% 
Level 2 High 2794 21 52% 2757 16 59% 2820 14 56% 
Level 3 7185 19 59% 6177 15 58% 5924 11 60% 
Level 4 5489 17 63% 4852 13 60% 4843 8 61% 
Level 5 2241 15 54% 2117 10 52% 2230 4 57% 
Overall 23470 58% 22617 56% 23548 58% 
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Table 5 (continued). 


2019 Grade Level 


Grade 8 Grade 9 Grade 10 

2017 Achievement Level N Target | Percentage N Target | Percentage N Target | Percentage 
Level 1 Low 1015 27 62% 945 22 58% 3035 18 59% 
Level 1 Middle 1160 23 54% 1267 18 57% 3492 15 50% 
Level 1 High 2721 19 53% 3152 15 54% 7531 12 52% 
Level 2 Low 2540 17 55% 2385 13 56% 4856 10 53% 
Level 2 High 2903 15 58% 2385 12 57% 4253 9 55% 
Level 3 5084 13 60% 4816 10 58% 7110 7 62% 
Level 4 5202 10 64% 4373 60% 4458 5 62% 
Level 5 1936 7 53% 2666 55% 2709 2 66% 
Overall 22561 
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